Enhancing the security of patients' portals and websites by detecting malicious web crawlers using machine learning techniques

Introduction: There is increasing demand for access to medical information via patients' portals. However, one of the challenges towards widespread utilisation of such service is maintaining the security of those portals. Recent reports show an alarming increase in cyber-attacks using crawlers. These software programs crawl web pages and are capable of executing various commands such as attacking web servers, cracking passwords, harvesting users' personal information, and testing the vulnerability of servers. The aim of this research is to develop a new effective model for detecting malicious crawlers based on their navigational behavior using machine-learning techniques.

Method: In this research, different methods of crawler detection were investigated. Log files of a sample of compromised web sites were analysed and the best features for the detection of crawlers were extracted. Then after testing and comparing several machine learning algorithms including Support Vector Machine (SVM), Bayesian Network and Decision Tree, the best model was developed using the most appropriate features and its accuracy was evaluated.

Results: Our analysis showed the SVM-based models can yield higher accuracy (f-measure = 0.97) comparing to Bayesian Network (f-measure = 0.88) and Decision Tree (f-measure = 0.95) and artificial neural network (ANN) (f-measure = 0.87)for detecting malicious crawlers. However, extracting proper features can increase the performance of the SVM (f-measure = 0.98), the Bayesian network (f-measure = 0.94) and the Decision Tree (f-measure = 0.96) and ANN (f-measure = 0.92).

Conclusion: Security concerns are among the potential barriers to widespread utilisation of patient portals. Machine learning algorithms can be accurately used to detect malicious crawlers and enhance the security of sensitive patients' information. Selecting appropriate features for the development of these algorithms can remarkably increase their accuracy.